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14. VALIDATION AND UPGRADING OF PHYSICALLY BASED 

MATHEMATICAL MODELS 


RONALD DU VAL 


I appreciate this opportunity to talk to this select 
group about these issues. The subject is validation and 
upgrading of physically based mathematical models. 

There are a lot of terms that are going to have to be 
defined. 

The previous speaker discussed validation from a 
totally different standpoint from the one I am going to 
address. He was looking at total validation of the simula- 
tion complex, which involves the motion-based system, 
the visual system, the transport delays — everything that 
affects the way a pilot perceives what is going on in the 
simulator. The starting point for all of these issues, how- 
ever, is the mathematical model that drives all of these 
systems. And it is very difficult to determine what consti- 
tutes validity in terms of visual display or what constitutes 
validity in terms of motion-based display. 

On the other hand, the determination of what consti- 
tutes validity in terms of a mathematical model is very 
straightforward: model validation is a systematic proce- 
dure for testing and modifying a simulation mathematical 
model to achieve the required level of fidelity in matching 
experimental data. So as a starting point in determining 
validation of an entire simulation complex, it makes sense 
to at least make sure the mathematical model on a stand- 
alone basis can be validated and then to go on and use the 
more subjective criteria he recommended for validation of 
the entire complex. So I am defining validation strictly 
from a standpoint of making sure the mathematical model 
that drives these systems has acceptable fidelity. 

The steps in validating a mathematical model are as 
follows: 

1 . Establish acceptance criteria 

2. Conduct flight tests and collect data 

3. Conduct simulation tests and compare results 

4. Analyzes discrepancies that exceed acceptance 
limits 

5. Modify the mathematical model to reduce dis- 
crepancies so they are within acceptance criteria limits 


I will go through each of these in more detail. The first 
step — the previous speaker made this point as well — is to 
establish the acceptance criteria. And that is very critical. 

It drives everything else from there on down. Once you 
have determined what is important to the missions you are 
trying to accomplish, then you can establish criteria to val- 
idate the model against those missions and then you can 
perform the rest of these activities: to conduct the neces- 
sary flight tests, and collect the data as a basis of compari- 
son; to perform simulation tests in an appropriate fashion 
to run comparisons with the experimental data; to analyze 
any discrepancies between the simulation results and the 
flight-test results; and, when those discrepancies exceed 
the acceptance criteria limits, to modify the mathematical 
model to bring those discrepancies within acceptable lim- 
its. The latter is, of course, the most difficult task. 

Let’s start with acceptance criteria, the first part of 
the procedure (table 1). I am going to define validation in 
two different ways: functional validation and physical 
validation. To begin with, functional validation, or accep- 
tance criteria to determine functional fidelity, basically 
requires fidelity of pilot cues. What you are trying to do is 
to make sure that what the pilot sees is an accurate repre- 
sentation of the input/out relationships of the aircraft. You 
don’t care what is going on inside the mathematical 
model. It is a black box. All you are really interested in is 
that given the right input you are getting the right output. 
That is functional fidelity. And this, of course, is the pri- 
mary way in which current training simulators are evalu- 
ated, on a functional basis. 

The kinds of criteria that are used for functional vali- 
dation are based on the effect, not on the cause. The 
response is being validated, not what is producing the 
response. The three classic criteria are trim, stability, and 
dynamic response. Regarding trim, you usually character- 
ize the control settings required to trim the aircraft at dif- 
ferent flight conditions. Often, stability is not specifically 
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Table 1. Acceptance criteria: functional fidelity 


1 . Requires fidelity of pilot cues 

2. Functional criteria (validate effect) 

Trim 

Stability 

Response 

3. Tuning factors: empirical coefficients 

4. Scope of validation: validation at system level 

5. Bandwidth of validation: limited to handling-qualities range 

6. Amplitude of validation: limited to linear range 


used as a criterion in the training industry. It is somewhat 
inherent in the response measurements, but stability char- 
acteristics could be prescribed either in the frequency 
domain or in the time domain. For the frequency domain, 
the phase or gain margins can be specified; for the time 
domain, the eigenvalues or eigenvectors can be specified. 
The dynamic response of the actual test vehicle can be 
compared both in the time and the frequency domains 
with similar responses for the simulation to determine 
whether the response is correct. This is often used in the 
training industry, at least in terms of time-domain 
responses. There is very little in the way of frequency- 
domain criteria that is being used right now f or v alidation. 

As far as the training industry is concerned, one of 
the major problems with the current acceptance criteria 
that have been established is that there is no attempt to 
specify how the manufacturer can tune the mathematical 
model to meet the acceptance criteria. The manufacturer 
basically has carte bl anche to do whatever he needs to in 
order to meet those acceptance criteria. And typically 
what happens is the manufacturer will add empirical coef- 
ficients at appropriate places in the simulation that make it 
very easy to tune in order to satisfy the acceptance 
criteria. 

I have see n a nu mber of cases in which scale factors 
and biases have been added to aerodynamic forces and 
moments. It is nonphysical, but it accomplishes the job of 
satisfying the specific test criteria. The problem with this 
kind of manipulation is that because it is done totally 
empirically, it satisfies the criteria at the test points but 
there is no guarantee that it is going to give a realistic 
response outside the test points or between test points. 
Basically, the test curve that the government gives you to 
satisfy is being fitted, and you can’t be assured that it is 
going to really represent the correct aircraft response. The 
other issues associated with the acceptance criteria are the 


scope of the validation. By this I meanis if end-to-end 
validation of the total aircraft that is of concern or is it the 
subsystems and their independent validation. 

Typically, right now validation is performed a t th e 
system level only, and it is based strictly on the six- 
degree-of-freedom aircraft rigid-body motion. If that is 
accomplished, the basic idea is that that is what the pilot 
sees, that is what the pilot perceives; there is no reason to 
carry validation into any more depth than that. The prob- 
lem with that, as we will shall see, is drat It allow s t he 
manufacturer to tweak a subsystem, the tweaking of 
which may be totally inappropriate, in order to get the 
total response correct. If the rotor model isn’t right, he 
may alter the control system in order to give the net 
response that is desired. By allowing vaJidatjon^af the 
global level, the manufacturer is given a lot of leeway In 
adjusting individual components, which in turn eliminates 
interchangeability and modularity of the resulting 
simulation. 

Another major issue is the bandwidth of the valida- 
tion, that is, the frequency to which the simulation must 
be accurate (table 1). And typically there have not been a 
lot of frequency-response criteria associated with training 
simulators. This is a major problem. The way in which it 
is evaluated, though, does predominantly limit th e band- 
width to the handling-qualities range, which again 
assumes that that is all the pilot is going to see andall he 
cares about. 

The last acceptance criterion, which is a really impor- 
tant issue, is the amplitude of the validation (table I). 
Typically, people will limit the periurbattqns jn Jhejjpear 
range. Validating the model when it is driven into its non- 
- linear range is a much more difficult job. There are virtu- 
ally no acceptance test criteria that enforce driving the 
model into the nonlinear range to see if it is accurately 
represented. What you end up with is a training simulator 
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that has been validated only in the middle of the envelope 
for mild maneuvering. If a pilot maneuvers it aggressively 
or flies to the edge of the envelope, the simulation and 
that environment based on these validation criteria have 
not been validated. And that is precisely where simulation 
should be particularly valuable, in conditions in which a 
pilot would not want to fly a real aircraft. That is typically 
not addressed in the validation criteria. 

Let’s take the other alternative, which is physical 
fidelity (table 2). By this I mean we are requiring that the 
mathematical representations of the physical phenomena 
in the simulation be correct. Instead of looking at the sim- 
ulation as a black box where all you are interested in is 
proper end-to-end response, you are going to look at the 
way in which the phenomena are modeled and try to vali- 
date it to that level. This is typically done in engineering 
simulators. The main reason it has not been used in train- 
ing simulators is because it is a much more difficult pro- 
cess, much more costly to do and to validate, and, ulti- 
mately, because it is very difficult to perform in real time, 
which is required for training in real-time simulations. 
What is happening right now, however, is that with the 
advent of parallel processing technology and modem 
high-speed computers, we can take physically based 
models and perform real-time simulation with them. 

Computer technology has been developed to the point 
where we can start using physically based models for real- 
time training applications. As a result, we need to look at 
what the advantages are of this kind of modeling to the 
training industry. Again, the acceptance criteria in a phys- 
ically based model are to validate the cause rather than the 
effect. Here what you are going to try is to compare 
applied loads for accelerations of the vehicle for given 
flight conditions. The way in which the model is tuned is 
much more restrictive than it is in a functional model. The 
only way the contractor is allowed to modify the system 
is to modify the structure of the mathematical model, in a 


physically meaningful manner, or to change physically 
meaningful parameters, not empirical coefficients. So it 
tremendously complicates the process of tuning the simu- 
lation to match the acceptance criteria. 

The scope of the validation is another important 
issue. Now we are talking about validating the system at 
the subsystem level. It is not acceptable to think of this as 
just a black box — that as long as the right response is 
obtained, we don’t care what goes on inside. You are now 
going to break the total model down to a main-rotor mod- 
ule, a tail-rotor module, horizontal stabilizers, and 
engines. Each of the components is going to be separately 
validated against independent test criteria so the control 
system can no longer be altered to make up for problems 
in the rotor model. The bandwidth of the validation now 
has to be significantly increased. And it has to be 
expanded to include the bandwidth of all modeled degrees 
of freedom in the system. If the subsystems are going to 
be validated with physically based models, it is necessary 
that the degrees of freedom of all the physically based 
models in the system be exercised. Of course, it is neces- 
sary to be able to excite it throughout the range, to be able 
to go into the nonlinear region and validate it there. 

One of the benefits of going to physically based mod- 
els is that it should make it possible to achieve global 
fidelity of the mathematical model; that is, you should be 
able to drive it to the edge of the envelope, fly it with 
aggressive maneuvering, and really use it as it should be 
used, as a tool for training a pilot in dangerous flying 
activities, those he could never achieve or even come 
close to, safely, in an aircraft. 

The third item on the list was flight test and data 
acquisition (table 3). These have to be geared to the 
acceptance criteria. Once the acceptance criteria are estab- 
lished, data must be collected to support the performance 
of this acceptance test. What is done is to collect data 
associated with functional validation, trim data, stability 


Table 2. Acceptance criteria: physical fidelity 


1 . Requires fidelity of mathematical representation of physical phenomena 

2. Physical criteria (validate cause): applied loads/acceleration 

3. Tuning factors 

Model structure 

Physically meaningful parameters 

4. Scope of validation: validation at subsystem level 

5. Bandwidth of validation: includes bandwidth of all modeled degrees of freedom 

6. Amplitude of validation: excites nonlinear range 
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data, response data; typically this is limited to the airframe 
rigid-body motion. 

Physical validation is a much more difficult problem. 
In order to isolate subsystems for independent validation, 
it must be possible to collect boundary data at each of the 
subsystems. For example, the reaction loads between the 
rotor and the fuselage must be measured so that the rotor 
can be isolated from the fuselage motion and validated as 
an independent subsystem. Typically, therefore, it must be 
possible to collect load data at the subsystem interface and 
to be able to collect acceleration rate and displacement 
data at subsystems. As a result, it is a much more difficult 
data-collection task. 

The way in which this is commonly performed, or 
can be performed, is to use redundant sensors and kine- 
matic constraints to eliminate the instrument, calibration, 
and procedure errors that are encountered. Too often raw 
test data with no cross-checking are used for acceptance 
test criteria. Our experience has been that such data are 
fraught with calibration errors and procedure errors. There 
are too many good ways available for doing consistency 


testing, kinematic cross-testing, for this to be the case. 

This should be used to ensure that you have the right 
experimental data to form the basis of the acceptance 
criteria. 

The mass properties and the sensor geometry must be 
documented. It must be possible to perform maneuvers 
that span the bandwidth and amplitude of the validation 
criteria. For the closed-loop simulation, here for the simu- 
lation tests, there are two approaches. The purpose of the 
closed-loop simulation is basically to initialize the simula- 
tion to the starting test conditions, drive it with test con- 
trol inputs, and then compare its response with the 
dynamic response of the test (table 4). 

This is the way in which it is ordinarily done. The 
advantage is that it is simple to implement and requires 
minimal sensor data. The disadvantage is that you have a 
cumulative buildup of error and you cannot isolate subsys- 
tems because of the coupling between them. The open- 
loop approach to testing the simulation is to disable the 
airframe rigid-body motion and drive the simulation with 
the control inputs and the rigid-body motion that has been 


Table 3. Flight test and data collection 


1. Functional validation 

Collect trim, stability, and response data for airframe rigid body degrees of freedom 

2. Physical validation 

Collect loads data at subsystem interfaces and acceleration, rate, and displacement data at subsystems 

3. Perform data consistency tests with redundant sensors and kinematic constraints to eliminate instrument 
calibration errors and procedural errors 

4. Document mass properties, sensor geometry, and atmospheric conditions during tests 

g^Performmaneu^ bandwidth and amj)liUKkyj>fj|g^ 


Table 4. Conduct simulation tests and compare results 



Closed-loop simulation 


1 . Method 

Initialize simulation to starting test condition 
Drive simulation with test control inputs 

Compare dynamic response of simulation to dynamic response of test 

2. Advantages 

Simple to implement 
Requires minimal sensor data 

3. Disadvantages 

Cumulative error build up due to closed-loop integration limits validity of comparison 
Coupling between dynamic subsystems limits ability to isolate discrepandM^__^_ 


126 


Physically Based Mathematical Modes 


determined from the test data (table 5). So what you are 
really doing now is driving the simulation on in a dynamic 
wind-tunnel mode and looking at the loads that are pro- 
duced along the same flight trajectory that the aircraft 
produced. You compare these loads with those obtained 
from the flight to validate the model. The advantage is 
that it eliminates cumulative error buildup and it allows 
the subsystems to be validated independently. The disad- 
vantage is that it is much more difficult to implement, and 
more expensive data are required to isolate the loads at the 
subsystems. 

For the analysis and modification methods there are 
two primary objectives: model structure has to be estab- 
lished and the parameters have to be modified (table 6). 
And the kinds of modifications you will typically have to 
make are to add coupling, higher-order dynamics, and 
nonlinearities. 

The parameter identification method used for linear- 
parameter dependency can be regression. The more diffi- 
cult problem of nonlinear dependencies would require an 


output-error approach. The point I have been making all 
along is that training simulators are functionally validated. 
The validation is performed at the system level with the 
rigid-body airframe response as the validation criterion. 
Satisfaction of this criterion is achieved by tuning empiri- 
cal coefficients. The result is a model tuned for specific 
conditions that has been validated only for bandwidth 
low-amplitude maneuvers (table 7). 

The bottom line is that validation requirements drive 
the modeling sophistication (table 8). You get what you 
ask for. And the simulation manufacturers will not pro- 
duce the physically based simulation if the validation 
requirements are functionally based. For example, rotor- 
map models are functional approximations to the blade- 
elements model; they satisfy acceptance test criteria as 
currently specified. However, you could specify criteria in 
a form such that contractors would have to go to a blade - 
elements model in order to achieve your requirements. In 
conclusion, what I think is really needed is a standard for 
rotorcraft validation that in a sense is like the standard that 


Table 5. Conduct simulation tests and compare results 


Open-Loop simulation 

1 . Method 

Disable integration of airframe rigid-body motion in simulation 
Drive simulation with control inputs and rigid-body motion from test data 
Compare loads/accelerations of simulation with test data 

2. Advantages 

Eliminates cumulative error build up due to integration of airframe states 
Allows subsystems to be isolated and validated independently 

3. Disadvantages 

Implementation of simulation run is more difficult 
More extensive test data are required to isolated loads at subsystems 


Table 6. Analysis/modification methods 


Model structure determination 


1. Correlate errors to states and controls for nominal parameter values 

Statistical correlation of error 
Frequency response of error 

2. Postulate modification to model structure 

Additional coupling 
Higher-order dynamics 
Nonlinearities 

3. Repeat comparison step and iterate until error can be sufficiently limited by reasonable parameter changes 
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Table 7. Problems with current validation approach 


1. Validation is only guaranteed in vicinity of test points 

2. Low-bandwidth validation does not support aggressive maneuvering, high-speed flight, or high-gain controllers 

3. Low-amplitude (linear) validation does not support aggressive or edge-of-the-envelope maneuvers 

4. Lack of subsystem validationjdr^i nates modularity and interchangeabilityjr^ 


Table 8. Validation requirements drive modeling sophistication 


You get what you ask for 

Simulation manufacturers will not produce physically based simulations if the validation requirements are functional 
Example: 

Rotormap models are functional approximations to the physically based blade-element model 

They satisfy trim and stability requirements and low-bandwidth response requirements for function validation 
They will not satisfy a validation criteria that specifies rotor impedance (rotor load frequency response to hub 
acceleration) . — : -s 


Table 9. Rotorcraft validation standard 


A standard for rotorcraft validation is required that will address the following: 

1 . Acceptance criteria versus simulator mission requirements 

2. Flight-test procedures and instrumentation versus acceptance test criteria 

3. Generation of simulation data and comparison with flight data 

4. Model structure determination and parameter identification methods for reducing errors to specified limits 

5. Acceptable physically based_parametere_forjuningjind[theh 


we are addressing here this week for simulation qualifica- 
tions (table 9). It could be either a part of the simulation 
qualifications or be detailed enough to require a separate 
specification. 

We have to define the acce ptance crite ria as a func- 
tion of the mission requirements. We have to determine 
flight-test procedures and instrumentations in order to be 
able to implement acceptance criteria. We have to be able 
to generate the simulation data and compare them with 
flight data in a systematic manner, apply modern tools for 
model structure determination, and parameter identifica- 
tion for achieving the criteria. Then we have to determine 
what physically based parameters are acceptable for tun- 
ing the simulation and what is their allowable range of 
validation. These are all terms that should be defined in a 
specification so that validation can be standardized. 

MR. WALKER: Since the interface between the sub- 
jective evaluator and the mathematical models is really 
the simulator that is provided by visual systems, motion 


bases, audio systems, and so on, how do you resolve the 
errors that may be introduced by these systems in the 
development of your valid ation? 

MR. DU VAL: I am referring strictly to the validation 
of the mathematical model; my contention is that you 
should not compensate for errors in these other systems 
by modifying the mathematical model; you should put in 
compensations for the systems, where they belong, that is, 
within the systems. 

MR. HAMPSON: I agree entirely with you. 1 do have 
some difficulty, though, with some of the comments you 
made with respect to tweaking the model. I don’t know if 
this is particularly a helicopter problem you are address- 
ing, but certainly with fixed-wing and also with the heli- 
copter models that are provided by the aircraft manufac- 
turer, we, as a simulator manufacturer, do not tweak the 
models. We identify the deficiencies and go back to the 
aircraft manufaciurer and tell him there is something 
wrong with his model or have him explain to us why we 
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have a problem. And I think that is the proper way to do 
things, rather than expecting the simulator manufacturer 
to tweak a model. 

It goes back to something I said yesterday, but in the 
helicopter world we rarely get a model from the manufac- 
turer of the aircraft. That is a significant issue, I think. 

MR. DU VAL: That is true. I haven’t really made the 
distinction of whether the mathematical model was gener- 
ated by the simulator manufacturer or the aircraft manu- 
facturer. The point is if the physically based mathematical 
model does not match the acceptance criteria, to add 
empirical parameters to make it match the criteria is not 
an appropriate solution, that it must be physically 
modified. 

MR. GALLOWAY: You mentioned that you get 
what you ask for. I would like to add the comment that 
you get what you pay for or are willing to pay for. How 


do I convince my Navy program managers to pay for the 
efforts you advocate for getting the data? 

Mr. DU VAL: The answer is modularity. You are 
going to pay for it in the short term, but you are going to 
get your money back in the long term. If you validate the 
subsystem models at the subsystem level, then you have 
interchangeability of mathematical models. You can plug 
in rotor models, you can build on them, because you vali- 
dated each of these components separately. It provides for 
the kind of modular interchangeable mathematical model- 
ing for simulation that we have been striving for. Once 
you have validated the basic component it is only a matter 
of changing the physical attributes to validate it with a dif- 
ferent aircraft. So even though it is more costly to do this 
up front, it is going to reduce the cost of validation on 
future simulation activities because you have building 
blocks you can work from. 
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